Less is More: Leveraging Belady’s Algorithm with Demand-based Learning

The increasing gap between fast data processing speed and long access latency has become the bottleneck of memory intensive workloads. One way to reduce average load to use latency is to minimize cache misses. While cold misses can be eliminated via prefetching schemes, conflict misses cannot be avoided without an efficient replacement policy. Given future cache access sequences, Belady’s algorithm decides whether a data block should get bypassed or cached and leads to the maximum number of cache hits in the most optimal scenario. However, Belady’s algorithm does not distinguish demand accesses such as loads and stores, which may cause pipeline stall due to data dependency, from writeback or prefetch accesses. Maximum number of cache hits does not guarantee highest performance. Therefore, we propose a high performance cache replacement policy which leverages key idea of Belady’s algorithm but focuses on demand accesses that have direct impact on system performance.